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ABSTRACT 

This pap€E initiates the ABBA sfffljosium oa issues 
metloas in gualitatiff€ eiraliaation with a theorstical ovecTieif ana 
critique of the state of the avaluation field in the anitaa Statas^ 
SQin« of the basic theoretiaal ujiaexpiiiniiigs to a quaiitative approscb 
to ^^aluation axe pneaanted, rhis type of apprcacl ccapleatmts aafl 
CQffljeii^ates for thcmm aspects ot educational practice that 
coni^aatloDal appcoachea omit^ (EC) 
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OVEBV^IEW ^ CBiTrQUE OF EXISIING EVAIUATIQN PRACMCES 
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^5 id IS Unl^et^ity of Illlaoti at Urtans-^Chanpatgia 



We live la a cruel and gie^tly tolerafit society. It is cruel Im that fev perisna, 
few prog^wsp SBd em iMtitutlofla aM allewcd to bgwae aacs thaa our e^cpicta- 
tlotis of thaiQ, nia society is toleramt ©f all sorts of varlsty and of fats 
mllltocia af rays o£ being aucctsifiul* 

^ paopla decnaiid any justif Icattoa of that i^hich alcsady esfists* We haife a gEa^t 

^ rhetDrlc ©f aaecutitability ^ competitloii, payment for results, and "the proof Is 

^ in the pudding*" Bat thcs# In prlifll^ged poalttons avoid scrutiny* And only th€ 

CI imo^atl^^ mem called tip to denonetrnt^ their af fecttveness* 

uy 

Evaiuators are called xo tha acenei agents of ratteaality, planned chMtgap asd 
anliahtstiment , T\mi%B ia a pQSltlQfi oC privilege* Xhsy spsak cf social service, 
tsshalcal pracialoa^ a»d utllltarlaiitsra. They w^k hard; spall welli aad protect 
their cllantp tfca fiundlng agenay, and thal^ Tsseareh collaaguea , They aate cospl^ac, 
though seldom atibatrassiiis, questions. They are ©ff i^hmt Basil latftatain calls 
the new middle clasij controHing the vsox&m th^t control the ttachiaea that keep 
socle t3? orderly and predictive, deffaa to me^Lt and afficl^ncy as tliey see 

it* 

Some of my^ best friends are evaiuators.. They are good people^ lellevltig In all 
slttcmrlty that theirs is a helping proeaaslom, that they ar^ Inchiiig tip^ard 
truth--'lf onl^r paop»Xii i?o\ild pay atteattoti and leara ho^ to interpret the data. 

For all Itg eeiapleiilty, educational evaluation Is a va^y ordinary cog In the-ioelal, 
econoffliCj polltielaed machitiecy^ Here are soma current points of contaat. 

IjCQI. The big RBTs fo^ evalttatlop continue to expact that th€ evaluation methods 
will be those of soclai^starvey research and experiaentation avail thougli these 
methods deliberately attisapt to avoid subjeetlve jiidppieiEit and valutngi even though 
these n^thods are Intent upon eontribnting to the anderatandlng of education in 
geaeral rathei tliaa to tha uadarstandiag of the partianlar progtama studied or to 
partletilar pregtMs lllca thep^ 

l^m^ Harraised by critlalan of early efforts to preseiit unint arpretad datap 
Nstlonal Assesament eutgolag director. Stan ^Ataann tntarprftted the most recent 
decline oC science sqo^as to waan ttkt'seieftee teaching in ^erica was beconlng 
poorer, a ccnclusien ii*ich aay ba true bnt oas f or t?hi«h he had aelther eaplrieal 
nor expertantlal gronnda, The validity of the National AafeMs^t tests has not 

Vf^ been actdbllihad far "atata-of-thc-health*of-education" purjiosss or fo* any policy* 
S€ttln$ purposes, Fnrtherrorep no standardlged taste Jiava ever been shoTO to hove 

%^ statistLcally baaed ^altdtty for naitna dectslona about prograffis at the dlitrict, 
state, or federal lawl* , 

CP 



preieotftd at th« Aanual Meeting of the toettcaft Educatl»ttal Reseafch 
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Vim* With §ewm tw^-thirds of ths statei having aaadatedi some form of state 
assessmeiit— iEcliawtftg vlgofoua ef forts tn Mlchl|sn mad Ilorida for half a do^ea 
yeaTs--*the plun^a tiitD thts tjrpi of stats ccntrol c£ edijeatlon has le\raled off » 
fflQstLy bacLause tlie early iresults wer^ not fsu^id to ha useful, wete costly, and 
were €ven becciiiag politiGally fimbar^asalQg , 

ITMl, Tliousatid^ of dlstrlcti f iad cfcamaelves forced to tncraass subetantially 
thalr budgeti fcr data fflaaagmant to aaat the appsttte for reports from top 
paople ill tha district sup^rititendent' a office or in the state iuperlntendeftt 'i 
off 1^6 5 thus fuirtliar dep3patlng moneys for curttcular pregrama in a ttaa of rising 
CQits and flKed school incomes. 

ItEtU Raaantly Lovington High School dropped its niMbeTShIp in the North Cantral 
Asaoeiaticia^ gluing up its accr^dltatiottp and the Uaivixsity of Illiiioli Collage 
of Ediieatton withdraw tmrn. MCTI me^exshtp, thus laalng its NCATE accreditation. 
Institutional acaradltatloup based oti ialC-^atudy and obsarvatloYis of visit lag 
pears, is increasingly ehallingcd as c-mbwrnom^ and eostly. To© many faoulty 
meobers see the self^itudy as a tjffli to hoodwink adninistrators , site visitors, 
and thm public, nofti off ^hcn ere thouglit to utiderstafld the pyogrtos ^3n?ay, But, 
mattilys the ieil«-study is dlaappoititinj heeause It doesnft help to satlafy state 
or federal data requirements. 

ITEM. TTaitiers of ivaluation specialists mobilise for e^Epanded work. Their 
eatlmates of ne^d for aew taMipowr nuifce ring. Into zhm thotjaaBds. Collage depart- 
^ants are creatinij new couraesi organlsatloai smh m i^RA eKpand their 
ewaluatlca-^trnisilaig*- institute oifertnge; afW orgMilsatloag sy^ as the Evalaation 
Sacwork aad thi "Evaluation Research Society of .toirtca are aborniUigi as are sew 
perlodlaala sucli mB the Journal for the Stud y of Educatloiial Evaluation s the 
faeulty DaTelQp aaiit and Eval uatiom Nei/gpaper^ and Citf, Haifslettsr * Still fflany 
stgtis of a growth Industry'*- but moat ol it depiEds om t^hat the go-yerfraenta require, 

ITM. Oregon CQngrissiroman Edith GreeDe , oaaa a Isadar in the dsx^elopment of 
federal edtiestion programa ^ is s supporter no lengar , iargely because of the large 
proportioa &£ funds sklmed off for adalslstratlDflp reaaarch^ devalopmeiit , and 
Valuation* Countless little aenileonm^rcial Goapanlea and erasultaneiea have 
j^lQed the higgias^ AEE> Eaad, BBC, EEfmO, EXS^\ABT Ajeoclatea to conpete fisr 
the dollars « Still i in&st of the TOlddlemea are sshool-^ and unlversity--baaed 
spaeialistSt 

irMi In 1964, David 4usubel wrote about tha naeassity ol aedieal^educatlan aval-* 
laatora knotd^mg the conteat sAd laiuea of medieal aducatlorip Mmoat uq cm paid 
attantiou, tben or tiov* %exi quallficatioiia Cor ei^alnators ara animerated^ It 
is eonslderad nlca^btjt aot eaaentlaL — if the evaluatar has a working knowledge 
©f the subject matter. Psyeli^lQglsts are eacouragad to define and waluate the 
basic skills In reading; atatlaticisna ara enaouraged to analyie Title 1 progr^s* 
Croai^f^rtlll^ition la mot without iralue, cf eoursai but aost avaluatera are only 
sentlltarata In the fields they axe evaluating , particularly In Edueation Itself * 
trhey can write Qb^acclv^es, develop tastij design e^qjertoentap and read computer 
laedbacki but they caiinct spaak IcteLllgently of padagogys currlculuoi, epiatemology , 
©r achool-coMiuttity reLationSt 
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ITEU* rhe WatlOTal Science loucidation is assiduc^sLy ieawhlng for nm pMgTaM 
revi^^ Md evaluatloft procedures • Prevloui evaliiatlon projecta did not al^rt them 
to adverse public reaction to the NSF curriculum 'Msg a Course of Study J' -AlmDSt 
VLom oi thaic ev^aluatori hmve been questioning irhathec the original fiupdteg 
daei^laas Mde sense and tmy avaliiatioa reports haw oall^d fer nagafclve ^atieni 
RgP haa a hstd tJtae ai^lalnliis what It has hmti g^tcing for Its evaluation dcllar* 

The evaluation buaisaas saaj^ ta a growth Industry, btat ctartaioly one with groining 
pains. Tod much has taso €Kpectsd^ too miah has baaa proBiised* rornaall^tle 
avaluatlos atudiis hajva ssldoBa Fielded Inf ormatioa wrch Its cost. At b^st they 
aduc©t# a few ^v-aluatorg m%y then oake raaenmeadations that axe halpaui* But 
y^DU can't aeunt on It. 

rh© key mistake* 1 thitikg Is the asai^ptien that objective laferMtion can b« 
aggregated acroae large Duaibers of teachers Dr stttdeati tc provide a basdi for 
deolsioa-aalclng to people %?ho are aot personally acquainted with the programs * 

The key hope^ I thiiiltj la that suhieetlve ioCormatiMp baaad on kay tsauae , 
orlaiited to real prcblama la partlaular iituataonSj rlgorciasly croas^aicasliied^ 
i^lll beeooje a standard oflmring of evaluatloa stmdtas. 

Et sterna to ma to ba a matter of eplstesiologyi Hqw do w knw our prograM? 
What eounts as evld#mea that a program is working? On mmy occaslQna 1 hai^a 
auigasted that the beat praettcal knoifledge If direct parsDnal e^ertacice and 
that tha ffl&at ireaaomable tfclng an evaluator saw offer im vicarious tHperlanot 
throiigh portrayals mi caS€ atudles* Such data are particularly valuable if the 
decislOQS atout programs axe to be ttada in the Individual claasroosi* 

Et im too auch to hcpe that any ififormation can impcove tlie ^llity of a distant 
auperlateBdaat or a€imiisi«iter to tell a teaoher t#het to da or a purehasdni agent 
mmt to purchase, Stilly the eKpanding dlstributtoa of avtliorlty and the ittcreais 
tn avomle of Taspoiis.ibiaity comiflg from dlatrlat^ scat©, M.d federal ofjlces 
require that we k^ep trying,. 

It la mot reascaable to suppoai that m inprovad evaluatica tachaalogy will make 
aducatlon acre affective or ioiva soeiaty's problaas* quaatltative tachm0craey 
la not working In the public tflterast^ but It could be Isga of a bind on tlie amall 
buaiaesa of getting school taught and helping kids gat edtieated* 

We deader if there are nonquantltativa teehnlqma ifhteh wculd lead to a uoim 
realistic knowledgs of prograffls without taflatlng ths costs ^ the red tape, and 
the social constralmta of evaluation. 
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